< p >蜘蛛池搭建示意图主要包括以下几个部分:IP代理池、User-Agent池、URL队列、网页爬取组件和数据存储器。其中,IP代理池负责提供大量的代理IP地址,User-Agent池负责提供大量的User-Agent,URL队列负责存储待爬取的网页链接,网页爬取组件负责模拟蜘蛛爬取网页,数据存储器负责将爬取到的数据进行存储。
Copyright 1995 - . All rights reserved. The content (including but not limited to text, photo, multimedia information, etc) published in this site belongs to China Daily Information Co (CDIC). Without written authorization from CDIC, such content shall not be republished or used in any form. Note: Browsers with 1024*768 or higher resolution are suggested for this site.